Automated Conformance Monitoring: Exploring the Path to Industrial Adoption

Architectural decisions can be interpreted as structural and behavioral constraints that must be enforced in order to guarantee overarching qualities in a system. Enforcing those constraints in a fully automated way is often challenging and not well supported by current tools. Practitioners are reluctant to invest in new solutions because of the difficulty of estimating the cost-effectiveness of a new tool, scarce resources allocated to quality related activities and a general lack of expertise in the domain. In this paper we investigate the dynamics involved in choosing and adopting a new automated conformance checking solution within an industrial context. We analyze a series of interviews identifying the most relevant criteria that affect user’s decisions. We also report on multiple case studies, describing how those criteria were used to steer the actual evaluation, integration and operation of a prototypical solution. Our study shows the advantages of using a flexible and semi-formal language for describing relevant design concerns. We observe that assuming any pro-active behavior from end-users is often unrealistic and that developers need to be actively informed and motivated. We show that integration with established quality reporting infrastructure is a key strategy for minimizing the impact of a new tool on processes and roles.